Person clothing feature extraction device, person search device, and processing method thereof

ABSTRACT

A person&#39;s region is detected from input video of a surveillance camera; a person&#39;s direction in the person&#39;s region is determined; the separability of person&#39;s clothes is determined to generate clothing segment separation information; furthermore, clothing features representing visual features of person&#39;s clothes in the person&#39;s region are extracted in consideration of the person&#39;s direction and the clothing segment separation information. The person&#39;s direction is determined based on a person&#39;s face direction, person&#39;s motion, and clothing symmetry. The clothing segment separation information is generated based on analysis information regarding a geometrical shape of the person&#39;s region and visual segment information representing person&#39;s clothing segments which are visible based on the person&#39;s region and background prior information. A person is searched out based on a result of matching between a clothing query text, representing a type and a color of person&#39;s clothes, and the extracted person&#39;s clothing features.

This is a divisional application based upon U.S. patent application Ser.No. 13/501,647 filed Apr. 12, 2012, which is a National Stage ofInternational Application No. PCT/JP2010/067914 filed Oct. 13, 2010,claiming priority based on Japanese Patent Application No. 2009-239360filed Oct. 16, 2009, the contents of all of which are incorporatedherein by reference in their entirety.

TECHNICAL FIELD

The present invention relates to a person clothing feature extractiondevice that extracts person clothing features from input video.Additionally, the present invention relates to a person search devicethat searches for persons based on person clothing features extractedfrom input video. Furthermore, the present invention relates to a personclothing feature extraction processing method and a person searchprocessing method.

BACKGROUND ART

Conventionally, a variety of person search methods, surveillancesystems, and image storing/search systems has been developed. PLT 1disclosed a person search method that searches for persons based onstored video in a surveillance system. When recording video, thesurveillance system extracts information, regarding person's faces andclothes, so as to store information in a database. When searchingpersons, the surveillance system compares face/clothing features of animage given as a query with face/clothing features stored in thedatabase so as to determine whether or not they indicate the sameperson.

A person search system disclosed in PLT 1 will be described in detailwith reference to FIG. 10. FIG. 10 shows the constitution of a personsearch device, which includes face region detection/face featureextraction parts 1000, 1020, clothing region detection/clothing featureextraction parts 1010, 1030, a clothing feature database 1040, a facefeature database 1050, a face similarity calculation part 1060, aclothing similarity calculation part 1070, and a person identitydetermination part 1080.

The face region detection/face feature extraction part 1000 detects aface region so as to extract face features based on video captured bythe surveillance system, thus sending extracted face features to theface feature database 1050. The clothing region detection/clothingfeature extraction part 1010 detects a clothing region so as to extractclothing features based on video captured by the surveillance system,thus sending extracted clothing features to the clothing featuredatabase 1040. On the other hand, the face region detection/face featureextraction part 1020 detects a face region so as to extract facefeatures from a query image, thus sending query face features to theclothing similarity calculation part 1070. The face similaritycalculation part 1060 compares query face features, received from theface region detection/face feature extraction part 1020, with facefeatures stored in the face feature database 1050, thus calculating andsending a face similarity to the person identity determination part1080. The clothing similarity calculation part 1070 compares queryclothing features, received from the clothing region detection/clothingfeature extraction part 1030, with clothing features stored in theclothing feature database 1040, thus calculating and sending a clothingsimilarity to the person identity determination part 1080. The personidentity determination part 1080 determines person identity based on aface similarity calculated by the face similarity calculation part 1060and a clothing similarity calculated by the clothing similaritycalculation part 1070, thus producing a person search result.

Next, the operation of the person search device shown in FIG. 10 will bedescribed. First, video captured by the surveillance system is inputinto the face region detection/face feature extraction part 1000 and theclothing region detection/clothing feature extraction part 1010. Theface region detection/face feature extraction part 1000 detects a faceregion per each frame of input video so as to extract face features fromthe detected face region. Face features which are extracted from a faceregion detected from input video are stored in the face feature database1050.

The clothing region detection/clothing feature extraction part 1010detects a clothing region from input video so as to extract its visualfeatures, i.e. clothing features. Extracted clothing features are storedin the clothing feature database 1040.

In case of person searching, a query image is input into the face regiondetection/face feature extraction part 1020 and the clothing regiondetection/clothing feature extraction part 1030. The face regiondetection/face feature extraction part 1020 and the clothing regiondetection/clothing feature extraction part 1030 function similarly tothe face region detection/face feature extraction part 1000 and theclothing region detection/clothing feature extraction part 1010 so as toextract query face features and query clothing features.

The face similarity calculation part 1060 compares query face featureswith face features stored in the face feature database 1050 so as tocalculate a face similarity. On the other hand, the clothing similaritycalculation part 1070 compares query clothing features with clothingfeatures stored in the clothing feature database 1040 so as to calculatea clothing similarity. The person identity determination part 1080integrates a face similarity with a clothing similarity so as todetermine person identity, thus producing a person search result.

PLT 2 disclosed an image storing/search system that searches for imagedata including image features equivalent to color sensation languagesrepresenting coloring which humans may subjectively sense. Herein, it isnecessary to preset a correlation between color expressions, included inhuman natural languages, and images in a color space. Additionally, itis necessary to extract pixels from image data stored in a database,calculate similarities with color expressions, and save them in memory.Provided a color expression as a query, this system checks similaritiesof image data with color expressions so as to search and display imagedata with a high similarity.

CITATION LIST Patent Literature

-   PLT 1: Japanese Patent Application Publication No. 2009-199322-   PLT 2: Japanese Patent Application Publication No. 2009-3581

SUMMARY OF INVENTION Technical Problem

The person search system shown in FIG. 10 is designed to solely acceptquery images but unable to search for persons based on query texts. Thatis, this system extracts visual features (e.g. information regardingcolors and patterns of clothes) from query images so as to search forpersons, but this system is unable to convert query texts, representinglanguage expressions such as “red clothes”, into visual features andthereby search for persons. Additionally, this system does not considera person's direction when detecting a clothing region from input video,so that this system is unable to consider differences of manners howclothes are viewed due to differences of person's directions.Furthermore, this system extracts visual features from the entireclothing region, but this system is unable to reflect differences ofclothes between the upper part and the lower part of a person's bodyinto person searching when different visual features are found betweenthe upper part and the lower part of a person's body wearing “a whitejacket and blue trousers”. For this reason, even when a searcher intendsto search for a person who turns his/her body in a front direction, thissystem is likely to produce a search result indicating another person.Additionally, even when a searcher intends to search for a person basedon his/her clothes in the upper part of his/her body, this system islikely to search another person with a similarity of visual featuresregarding clothes worn by another body part other than the upper part ofhis/her body. As described above, the conventional person search systemmay produce person search results including numerous errors.

The image storing/search system disclosed in PLT 2 is able to search forpersons based on a query text representing a single color of clothessuch as “red clothes”, but this system is unable to search for personsby use of multiple colors since its query text is able to simplydesignate a single color. Similar to PLT 1, this system is unable toreflect differences of person's directions and differences of visualfeatures, found between the upper part and the lower part of a person'sbody, into person search results.

Solution to Problem

The present invention is made in consideration of the aforementionedcircumstances, wherein the present invention provides a person clothingfeature extraction device which is able to extract clothing features ofperson's clothes included in video.

Additionally, the present invention provides a person search devicewhich searches for persons upon comparison between a query text andperson's clothing features detected from video.

Furthermore, the present invention provides programs describing a personclothing feature extraction processing method and a person searchprocessing method.

A person clothing feature extraction device of the present inventionincludes a person region detection part that detects a person's regionfrom input video; a person direction determination part that determinesa person's direction in the person's region; a clothing segmentseparation part that determines the separability of person's clothes inthe person's region so as to produce clothing segment separationinformation; a clothing feature extraction part that extracts clothingfeatures representing visual features of person's clothes in theperson's region in consideration of the person's direction and theclothing segment separation information; and a clothing feature storagethat stores the extracted clothing features.

A person search device of the present invention includes a clothinginformation extraction part that extracts clothing feature parametersbased on a clothing query text representing a type and a color ofperson's clothes; a clothing feature query generation part thatgenerates a clothing feature query based on clothing feature parameters;a clothing feature matching part that compares the clothing featurequery with clothing features retrieved from a clothing feature storage,thus producing its matching result; and a person search part thatproduces a person search result based on the matching result.

A person clothing feature extraction method of the present inventionexecutes a person region detecting process for detecting a person'sregion from input video; a person direction determining process fordetermining a person's direction in the person's region; a personclothing segment separation process for determining the separability ofperson's clothes in the person's region so as to produce clothingsegment separation information; and a clothing feature extractingprocess for extracting clothing features representing visual features ofperson's clothes in the person's region in consideration of the person'sdirection and the clothing segment separation information.

A person search method of the present invention executes a clothinginformation extracting process for extracting clothing featureparameters based on a clothing query text representing a type and acolor of person's clothes; a clothing feature query generating processfor generating a clothing feature query based on clothing featureparameters; a clothing feature matching process for comparing theclothing feature query with clothing features retrieved from a clothingfeature storage; and a person search process for producing a personsearch result based on a matching result.

The present invention provides a program describing the foregoingclothing feature extraction method in a computer-readable/executableformat. Additionally, the present invention provides a programdescribing the foregoing person search method in acomputer-readable/executable format.

Advantageous Effects of Invention

The present invention aims to detect a person's region from videocaptured by a surveillance camera or the like, accurately extractclothing features of a person who exists in the person's region, andthereby produce a person search result precisely reflecting a searcher'sintention based on extracted clothing features of a person.

BRIEF DESCRIPTION OF DRAWINGS

[FIG. 1] A block diagram showing the constitution of a person clothingfeature extraction device according to an embodiment of the presentinvention.

[FIG. 2] A block diagram showing the constitution of a person searchdevice according to an embodiment of the present invention.

[FIG. 3] A block diagram showing the internal constitution of a persondirection determination part.

[FIG. 4] A block diagram showing the internal constitution of a clothingsegment separation part.

[FIG. 5] A flowchart showing the processing of the person clothingfeature extraction device.

[FIG. 6] An illustration indicative of an example of a storage format ofvisual features representing person clothing features.

[FIG. 7] A flowchart showing the processing of the person directiondetermination part.

[FIG. 8] A flowchart showing the processing of the clothing segmentseparation part.

[FIG. 9] A flowchart showing the processing of the person search device.

[FIG. 10] A block diagram showing the constitution of a conventionalperson search system.

DESCRIPTION OF EMBODIMENTS

An embodiment of the present invention will be described in detail withreference to the accompanying drawings.

FIG. 1 is a block diagram showing the constitution of a person clothingfeature extraction device according to the present embodiment. Theperson clothing feature extraction device includes a person regiondetection part 100, a person direction determination part 110, aclothing segment separation part 120, and a clothing feature extractionpart 130, and a clothing feature storage 140.

The person clothing feature extraction device is realized by installinga person clothing feature extraction program in a computer configured ofa CPU, a ROM, and a RAM. The person clothing feature extraction program(or an information collection program) can be stored in various storagemedia, or they can be transferred via communication media. Storage mediamay encompass flexible disks, hard disks, magnetic disks, magneto-opticdisks, CD-ROM, DVD, ROM cartridges, battery-backup RAM cartridges, flashmemory cartridges, and nonvolatile RAM cartridges. Communication mediamay encompass wired communication media such as telephone lines, radiocommunication media such as microwave lines, and the Internet.

The person region detection part 100 detects a person's region thatexists in input video. A person's region detected from input video issupplied to the person direction determination part 110, the clothingsegment separation part 120, and the clothing feature extraction part130. The person direction determination part 110 determines a person'sdirection in a person's region of input video so as to send it to theclothing feature extraction part 130. The clothing segment separationpart 120 determines whether or not clothes of a person who exists in aperson's region of input video can be separated into segments, thusproviding clothing segment separation information to the clothingfeature extraction part 130. Specifically, it calculates clothingsegment separation information based on background prior information anda person's region of input video so as to send clothing segmentseparation information to the clothing feature extraction part 130. Theclothing feature extraction part 130 extracts visual information ofperson's clothes based on a person's region of input video, a person'sdirection, and clothing segment separation information, thus sendingvisual information to the clothing feature storage 140. In other words,it extracts person's clothing features from a person's region of inputvideo, a person's direction, and clothing segment separationinformation, thus sending them to the clothing feature storage 140. Theclothing feature storage 140 receives and stores person's clothingfeatures from the clothing feature extraction part 130.

Next, the operation of the person clothing feature extraction deviceshown in FIG. 1 will be described in detail. The person region detectionpart 100 receives desired video as an image subjected to processing,wherein it can accept image data of a predetermined compression formator image data of a non-compression format after decoding. As thecompression format, it is possible to employ an MPEG-2 (Moving PictureExpert Group) format or an H.264 format (or an MPEG-4 format). Imagedata of a compression format are subjected to decoding and then receivedin units of frames or in units of fields. The following descriptionrefers to input video aggregated in units of frames, but it is possibleto achieve the same image processing by use of input video aggregated inunits of fields. As a color format of input video, it is possible toemploy a YUV format, an RGB format, or other expression formats of acolor space.

The person region detection part 100 detects a person's region per eachframe of input video. As a person region detection processing method, itis possible to employ various methods. For instance, a differencebetween input video and pre-captured background image (hereinafter,referred to as a “differential image”) is calculated and subjected toexecution of a threshold process, thus extracting only the person'sregion from input video. It is possible to extract a differential image,representing a difference between input video and background image, foreach frame. Alternatively, each frame is divided into a plurality ofsubdivisions, so that it is possible to extract a differential image foreach subdivision. That is, it is possible to make a decision as toexistence/nonexistence of an individual animal (i.e. an animal otherthan a person) for each subdivision of each frame of input video, andsubsequently calculate a difference between background image and videodemonstrating nonexistence of an individual animal, thus extracting aperson's region. Additionally, it is possible to uniformly set athreshold, which is used for a threshold process executed on adifferential image, over the entire screen. Alternatively, it ispossible to adaptively set a threshold over each area on a screen. Forinstance, it is possible to increase a threshold with respect to an areaundergoing high variations of information over time on a screen, whileit is possible to decrease a threshold with respect to a stable areaundergoing small variations of information over time on a screen.

Person's regions extracted from input video are grouped into neighboringareas, wherein an individual ID (identification information) is assignedto each group, so that a person's region corresponds to an areaspecified by each ID. As a depicting method of each person's region, itis possible to employ various methods. For instance, it is possible tocalculate mask information representing two-dimensional informationassigned with a specific value which differs from a value representingbackground of an area specified by each ID. The calculated person'sregion is sent to the person direction determination part 110 togetherwith input video.

The person direction determination part 110 determines a person'sdirection based on a person's region and input video. A person'sdirection is determined based on a face direction, a person's motiondirection, and symmetry of person's clothes. This is because a person'sdirection exhibits a high correlation to a face direction and a person'smotion. Generally speaking, clothes may frequently exhibit right/leftsymmetrical patterns; hence, symmetry of clothes can be used todetermine whether or not a person turns his/her body in a frontdirection. These pieces of information are used to determine thedirection of a person who exists in a person's region in input video.Details of information used for determining a person's direction will bedescribed later. It is unnecessary to use all the face direction, theperson's motion direction, and the symmetry of clothes in determining aperson's direction; hence, it is possible to determine a person'sdirection based on at least one of these pieces of information. Herein,a person's direction is calculated per each area specified by each ID.For instance, a person's direction is calculated with respect to threesegments, namely a front direction, a rear direction, and anindeterminate direction (i.e. an indeterminate direction of a person).The calculated person's direction is supplied to the closing featureextraction part 130. In this connection, a person's direction is notnecessarily limited to three segments, namely a front direction, a reardirection, and an indeterminate direction; hence, a person's directioncan be classified into four or more segments.

The clothing segment separation part 120 receives a person's region,background prior information, and input video. The clothing segmentseparation part 120 separates person's clothes into a plurality ofsegments based on these pieces of information.

The background prior information is information representing a mannerhow a person is viewed depending on a position in the background of aperson's region. For instance, when an obstacle such as a desk and ashelf exists in a viewing scope of a camera and comes in contact withthe lower end of a person's region, it is considered that a part of aperson (e.g. the upper part of a body) may be visible in the backgroundprior information. Thus, a floor which exists in a viewing scope of acamera is marked as background prior information indicating that theentirety of a person may be visible, whilst the upper end of an obstacleis marked as background prior information indicating that a part of aperson (e.g. the upper part of a body) may be visible. In case of asurveillance camera with a fixed viewing scope, it is necessary tocapture background prior information at once. As a method of capturingbackground prior information, a surveyor may manually mark backgroundprior information and capture the information. With a surveillancecamera whose viewing scope is adaptable to a plurality of fixedpositions, a surveyor may manually mark background prior information andcapture the information at each fixed position. With a surveillancecamera whose viewing scope is continuously varied, a surveyor maytemporarily mark background prior information and capture theinformation, so that background prior information will be automaticallychanged to follow the motion of a surveillance camera. Specifically, aconventionally-known feature point extraction method is adopted toautomatically extract feature points such as corners of a desk or ashelf, so that feature points which are moving in a viewing scope tofollow the motion of a camera are correlated to each other betweenframes, thus keeping track of the movement of a person's region in eachbackground prior information.

The clothing feature extraction part 130 extracts visual features in aperson's region with respect to each segment of a person based on inputvideo, a person's region of the person region detection part 100, andclothing segment separation information of the clothing segmentseparation part 120.

When clothing segment separation information represents separationbetween the upper part and the lower part of a person's image andindicates its separation position, for example, it is possible toextract visual features regarding the upper part of a body from theupper area above the separation position while extracting visualfeatures regarding the lower part of a body from the lower area belowthe separation position in a person's region. Alternatively, it ispossible to determine segments such as a face and legs of a person fromthe upper part and the lower part of a body in a person's region, thusextracting visual features precluding those segments. Thus, visualfeatures extracted from a person's region are correlated to segments ofa person. For instance, visual features regarding the upper part of abody are provided in combination with an index representing the upperpart of a person's body. Additionally, they can be provided togetherwith a person's direction of the person direction determination part110. In case of a person's direction corresponding to his/her frontdirection, for example, visual features are provided together with anindex representing the front direction. In case of a person's directioncorresponding to his/her rear direction (or his/her side direction),visual features are provided together with an index representing therear direction (or the side direction). In case of a person's directioncorresponding to an indeterminable direction, visual features areprovided together with an index representing the indeterminabledirection (e.g. an index having a specific value).

Visual features represent colors and textures of persons' clothes. Usingvisual features, expressed in an HSV color space, pixel data of aperson's region are converted into Hue, Saturation, and Value, which arefurther subjected to quantization to produce an HSV histogram. In caseof visual features representing designated colors for use in DominantColor Descriptor of MPEG-7 specified in ISO/IEC 15938-3, a person'sregion is divided into color divisions, so that a dominant color issearched through each division so as to determine visual features. Otherthan this method, it is possible to use various visual featuresrepresenting colors for use in a color layout of MPEG-7. When an edgehistogram is used as representation of visual features regardingpatterns, an edge is detected in each direction of a person's region soas to produce an edge histogram. In case of visual features based on aWavelet method, a person's region is subjected to Wavelet conversion toproduce Wavelet coefficients. Wavelet coefficients or statics (i.e. anaverage, variance, etc. among Wavelet coefficients in its directionalcomponent) are used as visual features. Moreover, it is possible to usevarious visual features regarding patterns for use in HomogeneousTexture of MPEG-7. In this connection, visual features do notnecessarily contain both of color elements and pattern elements; hence,they may contain one of color elements and pattern elements.Furthermore, visual features may contain elements other than colorelements and pattern elements.

Visual features regarding person's clothes, which are extracted by theforegoing method, are stored in the clothing feature storage 140 asclothing features. As a storage format of clothing features, it ispossible to employ various formats. For instance, input video is dividedinto temporal units with a fixed time lengths, so that clothing featuresare stored in files in temporal units. Alternatively, video of a shortrecording time is stored in files in units of images. FIG. 6 shows anexample of a storage format of visual features. Herein, headerinformation is followed by visual features which are sequentially storedwith respect to each person's region. A person's region ID, a clothingsegment index, a person direction index, color-related visual features,and pattern-related visual features are sequentially stored with respectto each person's region. In this connection, the storage format ofvisual features is not necessarily limited to the format of FIG. 6;hence, it is possible to employ any format which can univocally specifyeach person's region.

Next, the operation of the person clothing feature extraction deviceshown in FIG. 1 will be described in detail. FIG. 5 is a flowchartshowing the entire processing of the person clothing feature extractiondevice. First, the person region detection part 100 detects a person'sregion per each frame from input video (step S100). Next, the persondirection determination part 110 determines a person's direction in aperson's region (step S110). Details of this process will be describedlater. Next, the clothing segment separation part 120 separates person'sclothes into a plurality of segments (step S120). Details of thisprocess will be described later. Next, the clothing feature extractionpart 130 extracts person's clothing features (step S130). In thisconnection, steps S110 and S120 can be reversed in their order.

The person clothing feature extraction device extracts and storesclothing features based on a person's direction and separability ofclothing segments. For this reason, it is possible to provideinformation (i.e. clothing feature information) which makes it possibleto search for clothes having different visual features with respect toeach person's direction and each clothing segment.

Next, the operation of the person direction determination part 110 ofthe person clothing feature extraction device will be described indetail. FIG. 3 is a block diagram showing the internal constitution ofthe person direction determination part 110. The person directiondetermination part 110 includes a face direction determination part 300,a person motion analysis part 310, a clothing symmetry determinationpart 320, and an integrative direction determination part 330.

The face direction determination part 300 determines a person's facedirection based on input video, thus providing its determination resultto the integrative direction determination part 330. The person motionanalysis part 310 analyzes person's motion based on input video and aperson's region, thus providing its determination result to theintegrative direction determination part 330. The clothing symmetrydetermination part 320 determines clothing symmetry based on input videoand a person's region, thus providing its determination result to theintegrative direction determination part 330. The integrative directiondetermination part 330 determines a person's direction based on aperson's face direction, a person's motion, and clothing symmetry.

Next, the operation of the person direction determination part 110 willbe described in detail. The face direction determination part 300detects a person's face region per each frame of input video whileestimating a face direction. As a method of detecting a person's faceregion and estimating a face direction, it is possible to employ variousmethods. When a plurality of persons' faces is detected in each frame ofinput video, it is necessary to estimate each person's face direction.Information regarding a person's face direction is a set of facepositions and directions (particularly, left/right directions) which arecollected with respect to each person's face. If no person's face isdetected in input video, it is necessary to provide informationindicating that no person's face is detected. To detect a person's faceregion and estimate a face direction, it is necessary to calculate areliability representing detection/estimation certainty, which isattached to information regarding a person's face direction. A person'sface direction, which is determined as described above, is supplied tothe integrative direction determination part 330.

The person motion analysis part 310 analyzes movement of a person'sregion based on time-series information pertaining to input video and aperson's region. For instance, it is possible to estimate movement of aperson's region by detecting feature points per each frame in a person'sregion and tracking them between frames. Alternatively, it is possibleto estimate movement of a person's region by calculating a centroidpoint of a person's region per each frame and tracking its movement. Inthis case, it is possible to estimate movement of a person's regionbased on two frames which occur sequentially in time series.Alternatively, it is possible to estimate movement of a person's regionbased on numerous frames. With respect to relatively small movement of aperson's region, it is necessary to calculate an optical flow betweenframes, thus estimating movement of a person's region based on theoptical flow. In this case, it is possible to calculate an average amongoptical flows between pixels in a person's region and perform anonlinear static process such as median, thus estimating movement of aperson's region. The estimated movement of a person's region (i.e. aperson motion) is supplied to the integrative direction determinationpart 330.

The clothing symmetry determination part 320 determines clothingsymmetry based on input video and a person's region. As a method ofdetermining clothing symmetry, it is possible to propose variousmethods. For instance, it is possible to check whether or not a pixelfunction, which can be produced by scanning pixels in a person's regionin a horizontal direction, exhibits symmetry about an axis proximate tothe center of a person's region. Specifically, it is necessary tocalculate symmetry deviation according to Equation 1.

$\begin{matrix}{{D_{s}(y)} = {\min\limits_{u}\frac{\sum\limits_{x = 0}^{W}{{\begin{matrix}{{I\left( {{u - x},y} \right)} -} \\{I\left( {{u + x},y} \right)}\end{matrix}}^{2}{M\left( {{u - x},y} \right)}{M\left( {{u + x},y} \right)}}}{\sum\limits_{x = 0}^{W}{{M\left( {{u - x},y} \right)}{M\left( {{u + x},y} \right)}}}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Herein, I(x,y) denotes pixel data (i.e. a three-dimensional vector in acolor space R, G, B) at coordinates (x,y). M(x,y) denotes maskinformation representing a person's region, wherein it is set to “1”when coordinates (x,y) represent a person's region, otherwise, it is setto “0”. W is a constant, and u is set to a value representing proximityto the center of a person's region. Equation 1 produces D_(s)(y) pereach value of y as a minimum value of symmetry deviation when the centerof a person's region is moved (i.e. when a value of u is varied). Thesymmetry deviation D_(s)(y), which is calculated as described above, isaveraged between an upper end y=Y₀ and a lower end y=Y₁ in a person'sregion in accordance with Equation 2, thus calculating an averagedistortion of symmetry.

$\begin{matrix}{\overset{\_}{D_{s}} = {\frac{1}{Y_{1} - Y_{0}}{\sum\limits_{y = Y_{0}}^{Y_{1}}{D_{s}(y)}}}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

It is possible to presume the situation where uniform pixel data appearin a viewing scope so as to increase symmetry regardless of directions;hence, it is necessary to concurrently check flatness of pixel data,whereby it is necessary to determine a low reliability in calculatinghigh symmetry deviation when pixel data exhibit high flatness. For thisreason, it is necessary to calculate flatness of pixel data for eachscanning line in a horizontal direction in accordance with Equation 3(i.e. a left end x=X₀ and a right end x=X₁ in a person's region) andthen average it between the upper end y=Y₀ and the lower end y=Y₁ in aperson's region in accordance with Equation 4, thus calculating anaverage distortion of symmetry.

$\begin{matrix}{{D_{f}(y)} = {\sum\limits_{x = X_{0}}^{X_{1}}{\left( {{I\left( {x,y} \right)} - \overset{\_}{I\left( {x,y} \right)}} \right)^{2}{M\left( {x,y} \right)}}}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack \\{\overset{\_}{D_{f}} = {\frac{1}{Y_{1} - Y_{0}}{\sum\limits_{y = Y_{0}}^{Y_{1}}{D_{f}(y)}}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Herein, I(x,y) with bar denotes an average of pixel data which can beobtained by scanning in a horizontal direction while fixing y. D_(s)with bar and D_(f) with bar, which are calculated as described above,are supplied to the integrative direction determination part 330 asclothing symmetry. Alternatively, it is possible to provide functionalvalues regarding symmetry and flatness, which are expressed in Equation1 and Equation 3, as clothing symmetry.

The integrative direction determination part 330 determines a person'sintegrative direction based on a person's face direction, person'smotion, and clothing symmetry in a person's region. Herein, it ispossible to employ various methods. For instance, it is possible tocalculate scores in a front direction (hereinafter, referred to as“frontality scores”) with respect to a person's face direction, person'smotion, and clothing symmetry, so that a person's direction isdetermined by integrating those scores. In this case, it is possible todirectly use a person's face direction as a frontality score.

As to person's motion, it is possible to calculate the similaritybetween the calculated motion vector and a downward vector, thusestimating in which direction a person is currently moving (or walking).For instance, it is possible to calculate a cosine value between amotion vector and a downward vector, thus estimating a person'sdirection based on the cosine value. Herein, a cosine value of “−1” iscalculated with respect to a motion vector corresponding to an upwardvector. Specifically, a frontality score is calculated according toEquation 5. A frontality value with a high positive value is calculatedwith respect to a motion vector with a high correlation to a downwardvector. In contrast, a frontality score with a high negative value iscalculated with respect to a motion vector with a high correlation to anupward vector. Herein, a downward direction is defined as a positivedirection of a y-axis.

$\begin{matrix}{S_{m} = {{\frac{1}{\sqrt{V_{x}^{2} + V_{y}^{2}}}{\begin{pmatrix}V_{x} \\V_{y}\end{pmatrix} \cdot \begin{pmatrix}0 \\1\end{pmatrix}}} = \frac{V_{y}}{\sqrt{V_{x}^{2} + V_{y}^{2}}}}} & \left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack\end{matrix}$

Herein, it is possible to calculate a frontality score in considerationof scalar quantity of motion vector. When scalar quantity of motionvector is equal to or less than a threshold, Equation 5 may produce afrontality score of “0”.

Additionally, it is possible to calculate a frontality score based onclothing symmetry. That is, when the clothing symmetry determinationpart 320 provides clothing symmetry using D_(s) with bar and D_(f) withbar, it is possible to calculate a frontality score according toEquation 6.

S _(c) =g( D _(f) )s( D _(s) )  [Equation 6]

Herein, g(x) denotes a monotone nondecreasing function, while s(x)denotes a monotone nonincreasing function which produces “0” when x is alarge number. Alternatively, when the clothing symmetry determinationpart 320 provides clothing symmetry using D_(s)(y) and D_(f)(y), it ispossible to calculate a frontality score according to Equation 7.

$\begin{matrix}{S_{c} = {s\left( {\frac{1}{Y_{1} - Y_{0}}{\sum\limits_{y = Y_{0}}^{Y_{1}}{{g\left( {D_{f}(y)} \right)}{D_{s}(y)}}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 7} \right\rbrack\end{matrix}$

As described above, it is possible to determine a person's directionbased on frontality scores which are calculated with respect to aperson's face direction, person's motion, and clothing symmetry. Herein,it is necessary to calculate the sum or product of frontality scores,thus determining that a person is currently turning in a front directionwhen the sum/product is higher than a predetermined threshold.Alternatively, it is possible to create an identification system such asa neural network, which inputs frontality scores so as to output adetermination result regarding integrative frontality, by use of alearning function of input data, thus determining person's frontality.Herein, a person's direction is determined by classifying it into afront direction, a rear direction, and an indeterminable direction.

Next, the entire process of the person direction determination part 110shown in FIG. 3 will be described with reference to a flowchart shown inFIG. 7. First, the face direction determination part 300 determines aface direction so as to forward its determination result to theintegrative direction determination part 330 (step S300). Next, theperson motion analysis part 310 estimates movement of a person's regionso as to forward its estimation result to the integrative directiondetermination part 330 (step S310). Next, the clothing symmetrydetermination part 320 determines person's clothing symmetry so as toforward its determination result to the integrative directiondetermination part 330 (step S320). Thereafter, the integrativedirection determination part 330 determines a person's integrativedirection based on a person's face direction, person's motion, andclothing symmetry (step S330). In this connection, it is possible tochange the order of steps S300, S310, and S320.

Next, the clothing segment separation part 120 of the person clothingfeature extraction device shown in FIG. 1 will be described in detail.FIG. 4 is a block diagram showing the internal constitution of theclothing segment separation part 120. The clothing segment separationpart 120 includes a regional shape analysis part 400, a visible segmentdetermination part 410, and an integrative segment separation part 420.

The regional shape analysis part 400 analyzes a person's region so as togenerate shape analysis information and forward it to the integrativesegment separation part 420. The visible segment determination part 410generates visible segment information based on a person's region andbackground prior information so as to forward it to the integrativesegment separation part 420. The integrated segment separation part 420generates clothing segment separation information based on input video,the shape analysis information of the regional shape determination part400, and the visible segment information of the visible segmentdetermination part 410.

Next, the process of the clothing segment separation part 120 shown inFIG. 4 will be described. The following description presumes twosegments, i.e. the upper part and the lower part of a body, as clothingsegments so that clothing segment separation information is generated asinformation for separating them. The regional shape analysis part 400analyzes a geometrical shape of a person's region so as to generateshape analysis information for determining whether or not a person iscurrently standing or for determining whether or not the upper part of aperson is solely reflected into a viewing scope. For instance, it ispossible to presume a rectangular area surrounding a person's region andthereby calculate its aspect ratio, thus generating shape analysisinformation. The calculated shape analysis information is supplied tothe integrative segment separation part 420.

The visible segment determination part 410 determines whether or not aperson is separable into the upper part and the lower part of a bodybased on a person's region and background prior information.Specifically, it determines whether or not a person's entire body isvisible in a person's region based on the background prior informationlocated at the coordinates which are calculated with respect to thelower end of a person's region. In the case where the background priorinformation indicates that a person's entire body is visible, visiblesegment information representing it is generated. In the case where thebackground prior information indicates that the upper part (or the lowerpart) of a person is solely visible in a person's region, visiblesegment information representing it is generated. In other cases,visible segment information representing indetermination of visiblesegments is generated. Visible segment information is supplied to theintegrative segment separation part 420.

The integrative segment separation part 420 determines whether or notperson's clothing segments are separable based on shape analysisinformation and visible segment information. Specifically, it determineswhether or not a person falls within an appropriate range (i.e. anappropriate range inside a viewing scope) based on shape analysisinformation. For instance, when visible segment information indicatesthat the entire body of a person who is standing straightly within anappropriate range is visible, the integrative segment separation part420 determines that a person is separable into the upper part and thelower part of a body. On the other hand, when shape analysis informationindicates that the upper part of a person's body solely falls within anappropriate range and when visible segment information indicates thatthe upper part of a person's body is visible, the integrative segmentseparation part 420 determines that the upper part of a person's body issolely visible. When visible segment information indicatesindetermination of visible segments, the integrative segment separationpart 420 determines that a person cannot be separated into the upperpart and the lower part of a body. Alternatively, when shape analysisinformation indicates that a person who is standing straightly does notfall within an appropriate range or when it indicates that the upperpart of a person's body does not fall within an appropriate range, theintegrative segment separation part 420 determines that a person cannotbe separated into the upper part and the lower part of a body.

Upon determining that a person is separable into the upper part and thelower part of a body, the integrative segment separation part 420calculates its separation position as well. As a method of calculating aseparation position, it is possible to propose various methods. Forinstance, it is possible to scan pixel data of a person's region in ahorizontal direction in accordance with Equation 8, thus calculatingpixel function values projected in a y-axis direction.

$\begin{matrix}{{f(y)} = \frac{\sum\limits_{x = X_{0}}^{X_{1}}{{I\left( {x,y} \right)}{M\left( {x,y} \right)}}}{\sum\limits_{x = X_{0}}^{X_{1}}{M\left( {x,y} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 8} \right\rbrack\end{matrix}$

Herein, I(x,y) denotes pixel data (i.e. a three-dimensional vector in acolor space R, G, B) at coordinates (x,y), while M(x,y) denotes maskinformation of a person's region. A y-coordinate that greatly varies apixel function value f(y) of Equation 8 is calculated. For instance,Equation 9 is used to produce a maximum value among differences of pixelfunction values; hence, it is possible to determine that a person isseparable into the upper part and the lower part of a body when amaximum difference D₀ is higher than a threshold. When it is determinedthat a person is separable into the upper part and the lower part of abody, Equation 10 is used to calculate a y-coordinate value y₀ at thisstate. As described above, the integrative segment separation part 420stores the y-coordinate value y₀ at the foregoing state, in addition tothe determination result that a person is separable into the upper partand the lower part of a body, in clothing segment separationinformation.

$\begin{matrix}{D_{0} = {\max\limits_{v}\left( {{\sum\limits_{y = 1}^{B}{f\left( {v - y} \right)}} - {\sum\limits_{y = 1}^{B}{f\left( {v + y} \right)}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 9} \right\rbrack \\{y_{0} = {\underset{v}{argmax}\left( {{\sum\limits_{y = 1}^{B}{f\left( {v - y} \right)}} - {\sum\limits_{y = 1}^{B}{f\left( {v + y} \right)}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 10} \right\rbrack\end{matrix}$

In this connection, when a relatively small difference is found amongvisual features between the upper part and the lower part of person'sclothes, it is determined that a person's region is separable with apredetermined separation ratio, so that the separation ratio is storedin clothing segment separation information.

Next, the process of the clothing segment separation part 120 shown inFIG. 4 will be described with reference to a flowchart shown in FIG. 8.First, the regional shape analysis part 400 analyzes the shape of aperson's region so as to forward shape analysis information to theintegrative segment separation part 420 (step S400). Next, the visiblesegment determination part 410 determines visible segments, which arevisible in a viewing scope of a camera, based on a person's region andbackground prior information, thus providing visible segment informationto the integrative segment separation part 420 (step S410). Theintegrative segment separation part 420 executes the foregoingintegrative segment separation process so as to generate clothingsegment separation information (step S420). In this connection, it ispossible to change the order of steps S400 and S410.

Next, a person search device according to an embodiment of the presentinvention will be described in detail. FIG. 2 is a block diagram showingthe constitution of the person search device according to the presentembodiment. The person search device includes a clothing informationextraction part 200, a clothing feature query generation part 210, aclothing feature matching part 220, and a person search part 230 as wellas the clothing feature storage 140. The person search device shown inFIG. 2 does not include constituent elements of the person clothingfeature extraction device except for the clothing feature storage 140shown in FIG. 1. However, it is possible to combine the person searchdevice of FIG. 2 with the person clothing feature extraction device ofFIG. 1.

The functionality of the person search device is realized by installinga person search program in a computer which is configured of a CPU, aROM, and a RAM.

The clothing information extraction part 200 extracts words representingtypes of clothes and visual features based on clothing query texts, thusproducing clothing feature parameters. Specifically, it analyzes aclothing query text with reference to a clothing dictionary, thusproducing and providing clothing feature parameters to the clothingfeature query generation part 210. The clothing feature query generationpart 210 estimates clothing visual features based on clothing featureparameters, thus producing and providing a clothing feature query to theclothing feature matching part 220. The clothing feature storage 140stores person's clothing features which are extracted by means of theperson clothing feature extraction part shown in FIG. 1. Clothingfeatures are visual features regarding person's clothes which aregenerated based on input video, a person's region, and clothing segmentseparation information. It is possible to additionally include visualfeatured regarding clothes, which are extracted based on a person'sdirection, in clothing features. The clothing feature matching part 220compares a clothing feature query with clothing features stored in theclothing feature storage 140, thus providing its matching result to theperson search part 230. The person search part 230 integrates matchingresults produced by the clothing feature matching part 220, thusproducing a person search result.

Next, the operation of the person search device shown in FIG. 2 will bedescribed in detail. The clothing information extraction part 200extracts clothing feature parameters, representing types and colors ofclothes, with reference to a clothing dictionary based on a clothingquery text. The clothing dictionary stores clothing information, i.e.pixel data (e.g. RGB data or HSV data), in connection with wordsrepresenting various colors. Additionally, the clothing dictionary mayinclude information, as to whether types of clothes are related to theupper part or the lower part of a body, in clothing information. Theclothing information extraction part 200 analyzes a clothing query textwith reference to clothing information, which is registered with theclothing dictionary, thus generating clothing feature parameters.

Upon receiving a clothing query text representing “a white shirt, a bluejacket, and black trousers”, for example, the clothing informationextraction part 200 extracts from the clothing dictionary the clothinginformation that “shirt” and “jacket” are related to the upper part of abody while they are colored “white” and “blue” respectively.Additionally, the clothing information extraction part 200 extracts fromthe clothing dictionary the clothing information that “trousers” arerelated to the lower part of a body while it is colored “black”.Furthermore, the clothing information extraction part 200 determinesthat, when a shirt and a jacket are simultaneously worn by a person, thejacket appears in the front layer of wearing so that “blue” has a higherratio than “white” as the total color of the upper part of a body.Considering a fact that both the shirt and the jacket are visiblethrough observation of a person in his/her front direction whilst thejacket is solely visible through observation of a person in his/her reardirection, the clothing information extraction part 200 generatesdifferent color parameters with respect to the front side and the rearside in the upper part of a body. As to the lower part of a body, itgenerates a color parameter indicating that both the front side and therear side are colored “black”.

When a clothing query text includes a language representing ambiguitysuch as “roughly . . . ” and “a sort of . . . ” in Japanese, it isnecessary to describe a degree of ambiguity in a clothing featureparameter. When clothing features are displayed using a color histogram,for example, it is possible to adjust a spreading factor of the colorhistogram with a degree of ambiguity.

As described above, pixel data (e.g. RGB data or HSV data) regarding theupper part and the lower part of a body, the color ratio, the colorparameter, and the degree of ambiguity in color rendition are describedin clothing feature parameters, which are supplied to the clothingfeature query generation part 210.

The clothing feature query generation part 210 generates a clothingfeature query based on clothing feature parameters provided from theclothing information extraction part 200. When a color histogram is usedto represent clothing features, for example, it is necessary to create acolor histogram with a peak value corresponding to pixel data includedin clothing feature parameters, and a spreading factor dependent upon adegree of ambiguity in color rendition. Herein, the peak value of acolor histogram is adjusted in response to the color ratio.Additionally, it is necessary to create color histograms individuallywith respect to a front direction and a non-front direction of a personbecause a manner how colors of clothes are viewed depends on a person'sdirection (i.e. a decision whether or not a person is turning in his/herfront direction). Furthermore, it is necessary to create clothingfeature queries with respect to the upper part and the lower part of abody. Clothing feature queries, which are created as described above,are supplied to the clothing feature matching part 220.

The clothing feature matching part 220 compares a clothing featurequery, which is generated by the clothing feature query generation part210, with clothing features (i.e. clothing features of a personcurrently subjected to searching) stored in the clothing feature storage140, thus calculating matching scores. Comparison between a clothingfeature query and clothing features (which are retrieved from theclothing feature storage 140) can be carried out with respect to each ofclothing segments specified by clothing segment separation information.

The foregoing matching scores represent similarities of clothingfeatures; for example, they can be calculated using the scalar productof clothing feature vectors. Alternatively, it is possible to calculatedistances (or differences) between clothing features and subsequentlyconvert them into similarities of clothing features. That is, a distanced between clothing features is converted into a similarity S inaccordance with Equation 11.

$\begin{matrix}{S = \frac{1}{1 + d}} & \left\lbrack {{Equation}\mspace{14mu} 11} \right\rbrack\end{matrix}$

Since clothing features stored in the clothing feature storage 140 arestored in connection with a person's direction and clothing segmentseparation information, the clothing feature matching part 220 considersits information to compare a clothing feature query with retrievedclothing features. Specifically, when retrieved clothing features arecorrelated to front sides of persons, they should be compared with aclothing feature query pertaining to the front side of a person. Whenretrieved clothing features are correlated to non-front directions ofpersons, they should be compared with a clothing feature querypertaining to a non-front direction of a person. When retrieved clothingfeatures are correlated to indeterminable directions of persons, theyshould be compared with both the clothing feature queries pertaining toa front direction and a non-front direction of a person, so that abetter matching result will be selected.

When retrieved clothing features are correlated to the upper parts ofpersons, they should be compared with a clothing feature querypertaining to the upper part of a person's body in consideration ofclothing segment separation information. When retrieved clothingfeatures are correlated to the lower parts of persons, they should becompared with a clothing feature query pertaining to the lower part of aperson's body. On the other hand, when person's clothes cannot beseparated into the upper part and the lower part of a person's body, itis necessary to compare them with an integrative clothing feature querywhich is created by integrating clothing feature queries pertaining tothe upper part and the lower part of a person's body. When a colorhistogram is used to represent clothing features, for example, clothingfeatures pertaining to the upper part and the lower part of a person'sbody are added together, normalized as necessary, and then compared toretrieved clothing features.

When the upper part and the lower part of a person's body areconcurrently visible, the clothing feature matching part 220 is able toconcurrently produce matching results with respect to both the upperpart and the lower part of a person's body. In this case, it is possibleto determine a matching degree by using both the matching results ofclothing features pertaining to the upper part and the lower part of aperson's body. When one of the upper part and the lower part of aperson's body is visible, or when person's clothes cannot be separatedinto the upper part and the lower part of a person's body, the clothingfeature matching part 220 may solely produce one matching result. Inthis case, it is necessary to determine a matching degree by using onematching result.

As described above, the clothing feature matching part 220 generatesdifferent numbers of matching results depending on a manner how a personis visible. When one matching result is produced, a similarity ofclothing features at this state is directly used as an entire matchingresult. When a plurality of matching results pertaining to the upperpart and the lower part of a person's body or the like is produced, itis possible to calculate a plurality of similarities S₁, S₂ regardingclothing features. In this case, it is necessary to calculate anintegrative similarity S according to Equation 12.

S=h(S ₁ +S _(s))

Herein, h(x) denotes a monotone nondecreasing function, for example,which is expressed via Equation 12.

h(s)=√{square root over (s)}  [Equation 13]

Owing to the foregoing processes, it is possible to produce matchingresults, which are close to person's clothing features by intuition, byincreasing similarities which are calculated via integrative matching.Matching results of the clothing feature matching part 220 are suppliedto the person search part 230.

The person search part 230 produces person search results based onmatching results of the clothing feature matching part 220.Specifically, matching results are sorted in a descending order ofintegrative similarities so that they are provided as person searchresults. When the number of matching results included in a person searchresult is fixed to N, it is necessary to selectively provide N matchingresults in upper places in the descending order of integrativesimilarities.

Next, the operation of the person search device shown in FIG. 2 will bedescribed in detail with reference to a flowchart shown in FIG. 9.First, the clothing information extraction part 200 extracts clothingfeature parameters, corresponding to a clothing query text, with theclothing dictionary, thus forwarding them to the clothing feature querygeneration part 210 (step S200). Next, the clothing feature querygeneration part 210 generates a clothing feature query based on clothingfeature parameters, thus forwarding it to the clothing feature matchingpart 220 (step S210). Next, the clothing feature matching part 220compares a clothing feature query with retrieved clothing features (readfrom the clothing feature storage 140), thus forwarding its matchingresult to the person search part 230 (step S220). The person search part230 produces and forwards a person search result based on a clothingfeature query and retrieved clothing features (step S230).

The person search device shown in FIG. 2 achieves high-level personsearch processing based on a clothing query text, which is expressedusing natural languages such as Japanese and English, in considerationof a person's direction and clothing segment separation information. Inother words, the present embodiment is designed to input clothingfeatures of persons expressed in natural languages so as to search forpersons in consideration of differences of persons' directions andclothing features; hence, it is possible to produce person searchresults close to searcher's intention.

In this connection, the present invention is not necessarily limited tothe foregoing embodiment so that it allows for any changes within thescope of the invention defined by the appended claims. For instance, theclothing feature separation part 120 does not necessarily separateperson's clothing segments into two parts, i.e. the upper part and thelower part of a body, but it may separate them into other clothingsegments such as shoes and hats.

INDUSTRIAL APPLICABILITY

The present invention is designed to detect a person's region within theviewing scope of a surveillance camera, extract person's clothingfeatures included in the person's region, and thereby search for aperson with a high accuracy with reference to a database; hence, thepresent invention is applicable to security usages in publicorganizations and private companies.

REFERENCE SIGNS LIST

-   100 person region detection part-   110 person direction determination part-   120 clothing segment separation part-   130 clothing feature extraction part-   140 clothing feature storage-   200 clothing information extraction part-   210 clothing feature query generation part-   220 clothing feature matching part-   230 person search part-   300 face direction determination part-   310 person motion analysis part-   320 clothing feature symmetry determination part-   330 integrative direction determination part-   400 regional shape analysis part-   410 visible segment determination part-   420 integrative segment separation part-   1000 face region detection/face feature extraction part-   1010 clothing region detection/clothing feature extraction part-   1020 face region detection/face feature extraction part-   1030 clothing region detection/clothing feature extraction part-   1040 clothing feature database (DB)-   1050 face feature database (DB)-   1060 face similarity calculation part-   1070 clothing similarity calculation part-   1080 person identity determination part

1. A person clothing feature extraction device comprising: a personregion detection part that detects a person's region from input video; aperson direction determination part that determines a person's directionin the person's region; a clothing segment separation part thatdetermines a separability of person's clothes in the person's region soas to produce clothing segment separation information reflecting anautomatic separability of clothing segments and a separable manner howclothing segments are separated; a clothing feature extraction part,considering the person's direction and the clothing segment separationinformation, which extracts clothing features representing visualfeatures of person's clothes in the person's region with respect to eachclothing segment when clothing segments are automatically separable butwhich extracts clothing features without separating them when clothingfeatures are not automatically separable; and a clothing feature storagethat stores the extracted clothing features.
 2. The person clothingfeature extraction device according to claim 1, wherein the persondirection determination part determines the person's direction based onat least one of a person's face direction, person's motion, and clothingsymmetry.
 3. The person clothing feature extraction device according toclaim 1, wherein the person's direction determined by the persondirection determination part indicates at least one of a frontdirection, a rear direction, and an indeterminable direction.
 4. Theperson clothing feature extraction device according to claim 1, whereinthe clothing segment separation part determines the separability ofperson's clothes based on the input video, the person's region, andbackground prior information.
 5. The person clothing feature extractiondevice according to claim 4, wherein the clothing segment separationpart includes a regional shape analysis part which analyzes a geometricshape of the person's region so as to produce shape analysisinformation, a visible segment determination part which produces visiblesegment information representing person's clothing segments visualizedbased on the person's region and the background prior information, andan integrative segment separation part which determines the separabilityof person's clothing segments based on the shape analysis informationand the visible segment information so as to produce the clothingsegment separation information.
 6. A person clothing feature extractionmethod comprising: detecting a person's region from input video;determining a person's direction in the person's region; determining aseparability of person's clothes in the person's region so as to produceclothing segment separation information; and extracting and storingclothing features representing visual features of person's clothes inthe person's region in consideration of the person's direction and theclothing segment separation information.
 7. The person clothing featureextraction method according to claim 6, wherein the person's directionis determined based on at least one of a person's face direction,person's motion, and clothing symmetry.
 8. The person clothing featureextraction method according to claim 6, wherein the person's directionindicates at least one of a front direction, a rear direction, and anindeterminable direction.
 9. The person clothing feature extractionmethod according to claim 6, wherein the separability of person'sclothes is determined based on the input video, the person's region, andbackground prior information.
 10. The person clothing feature extractionmethod according to claim 9, further comprising: analyzing a geometricshape of the person's region so as to produce shape analysisinformation; producing visible segment information representing person'sclothing segments which are visible based on the person's region and thebackground prior information; and determining the separability ofperson's clothing segments based on the shape analysis information andthe visible segment information so as to produce the clothing segmentseparation information.
 11. A program which describes the clothingfeature extraction method as defined in claim 6 in acomputer-readable/executable format.